fetch_dhan_data.py

Fetches full market data for all NSE stocks from Dhan ScanX API

Overview

The fetch_dhan_data.py script retrieves comprehensive market data for all NSE equity stocks (approximately 2,775 securities) in a single API call. This is the first script in the EDL Pipeline and creates the foundation for all subsequent data fetching operations.

Purpose

Fetches real-time market metrics including:

• Market cap, price, and valuation ratios (P/E, P/B, ROE, ROCE)
• Technical indicators (SMA, RSI, Moving Averages)
• Ownership data (Promoter holding, FII/DII percentages)
• Fundamental metrics (Revenue, Profit Margins, EPS)
• Security identifiers (ISIN, Symbol, Sid)

API Endpoint

URL · string · required

https://ow-scanx-analytics.dhan.co/customscan/fetchdt

Method · string · required

POST

Request Payload

{
  "data": {
    "type": "full", 
    "whichpage": "nse_total_market",
    "filters": [], 
    "sort": "Mcap", 
    "sorder": "desc",
    "count": 5000, 
    "page": 1
  }
}

Parameters

data.sort · string · default: "Mcap"

Field to sort results by (Market Cap)

data.sorder · string · default: "desc"

Sort order (desc or asc)

data.count · number · default: "5000"

Maximum number of records to fetch. Set to 5000 to ensure all ~2,775 stocks are returned in one call.

data.page · number · default: "1"

Page number for pagination

data.fields · array

List of fields to retrieve (see code example for full list)

data.params · array

Filter parameters:

• {"field": "OgInst", "op": "", "val": "ES"} - Equity Segment
• {"field": "Exch", "op": "", "val": "NSE"} - NSE Exchange

Output Files

dhan_data_response.json · array

Raw API response containing all market data for ~2,775 stocks. Each object includes fields like Sym, Isin, Mcap, Pe, Ltp, Volume, Sid, etc.

master_isin_map.json · array

Normalized mapping file used by all downstream scripts. Contains:

• Symbol - Stock symbol (e.g., “RELIANCE”)
• ISIN - International Securities Identification Number
• Name - Display name
• Sid - Security ID required for OHLCV and indicator APIs
• FnoFlag - F&O eligibility flag (1 = Yes, 0 = No)

Function Signature

def fetch_all_dhan_data():
    """
    Fetches all NSE stock data and creates master ISIN mapping.
    
    Returns:
        None (writes to files)
        
    Output Files:
        - dhan_data_response.json: Raw market data
        - master_isin_map.json: ISIN to Symbol mapping with Sid
    """

Dependencies

Python Packages · list

• requests - HTTP client
• json - JSON processing
• os - File path operations

Local Modules · list

• pipeline_utils.get_headers() - Returns standard API headers with rotating User-Agent

Prerequisites · list

None - This is the first script in the pipeline

Code Example

import requests
import json
import os
from pipeline_utils import get_headers

def fetch_all_dhan_data():
    url = "https://ow-scanx-analytics.dhan.co/customscan/fetchdt"
    
    BASE_DIR = os.path.dirname(os.path.abspath(__file__))
    output_file = os.path.join(BASE_DIR, "dhan_data_response.json")
    master_map_file = os.path.join(BASE_DIR, "master_isin_map.json")

    payload = {
        "data": {
            "sort": "Mcap",
            "sorder": "desc",
            "count": 5000,
            "fields": [
                "Isin", "DispSym", "Mcap", "Pe", "DivYeild", "Revenue", 
                "Year1RevenueGrowth", "NetProfitMargin", "YoYLastQtrlyProfitGrowth", 
                "EBIDTAMargin", "volume", "PricePerchng1year", "Sym", "Sid", "FnoFlag"
            ],
            "params": [
                {"field": "OgInst", "op": "", "val": "ES"},
                {"field": "Exch", "op": "", "val": "NSE"}
            ],
            "pgno": 0
        }
    }

    headers = get_headers(include_origin=True)

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()
    
    data = response.json()
    
    if 'data' in data and isinstance(data['data'], list):
        cleaned_data = data['data']
        
        # Save raw response
        with open(output_file, "w") as f:
            json.dump(cleaned_data, f, indent=4)
        print(f"Successfully fetched {len(cleaned_data)} items")
        
        # Create Master ISIN Map
        master_map = []
        for item in cleaned_data:
            if item.get('Sym') and item.get('Isin'):
                master_map.append({
                    "Symbol": item.get('Sym'),
                    "ISIN": item.get('Isin'),
                    "Name": item.get('DispSym'),
                    "Sid": item.get('Sid'),
                    "FnoFlag": item.get('FnoFlag', 0)
                })
        
        master_map.sort(key=lambda x: x['Symbol'])
        
        with open(master_map_file, "w") as f_map:
            json.dump(master_map, f_map, indent=4)
        print(f"Saved {len(master_map)} symbols to master_isin_map.json")

Usage

python3 fetch_dhan_data.py

Performance

• Execution Time: ~3-5 seconds
• API Calls: 1 request
• Output Size: ~5-8 MB (dhan_data_response.json)
• Concurrency: Not applicable (single request)

Notes

• Returns all ~2,775 NSE equity stocks in a single API call
• No pagination required due to high count: 5000 parameter
• master_isin_map.json is used by all subsequent fetching scripts
• The Sid field is critical for OHLCV and advanced indicator APIs